Papers with true positive rate
An Annotated Dataset of Errors in Premodern Greek and Baselines for Detecting Them (2025.findings-naacl)
Copied to clipboard
Creston Brooks, Johannes Haubold, Charlie Cowen-Breen, Jay White, Desmond DeVaul, Frederick Riemenschneider, Karthik R Narasimhan, Barbara Graziosi
| Challenge: | a dataset of real errors in premodern Greek is presented to improve error detection methods . scribal errors are more difficult to detect than print or digitization errors. |
| Approach: | They propose to annotate 1,000 words more likely to contain errors and annotated them as errors or not . they propose to evaluate new error detection methods that outperform other methods . |
| Outcome: | The proposed method outperforms all other methods, improving true positive rate by 5%. |
Towards Statistical Factuality Guarantee for Large Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Advancements in Large Vision-Language Models (LVLMs) have demonstrated impressive performance in image-conditioned text generation, but hallucinated outputs pose a major barrier to their use in safety-critical applications. |
| Approach: | They propose a conformal-prediction-based framework that achieves finite-sample distribution-free statistical guarantees to the factuality of LVLM output. |
| Outcome: | The proposed framework reduces the error rate of LLaVa-1.5 claims from 87.8% to 10.0% while ensuring that the output is accurate. |
Powerful Training-Free Membership Inference Against Fine-Tuned Autoregressive Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for auditing fine-tuned language models have limited detection rates . membership inference attacks aim to determine if a specific record was in a model's training set . |
| Approach: | They propose a membership inference attack that exploits memorization at error positions . EZ-MIA achieves 3.8 higher detection than previous state-of-the-art . |
| Outcome: | The proposed attack achieves 3.8 higher detection than previous state-of-the-art models . EZ-MIA achieves 8 higher detectability than prior work, requiring no model training . |
RevPRAG: Revealing Poisoning Attacks in Retrieval-Augmented Generation through LLM Activation Analysis (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) enriches the input to LLMs by retrieving information from the relevant knowledge database. |
| Approach: | They propose to use a knowledge database to enrich the input of LLMs by retrieving information from the relevant knowledge database. |
| Outcome: | The proposed approach can achieve 98% true positive rate while maintaining a false positive rate close to 1%. |
GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluation (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown consistent improvements across many tasks requiring natural language understanding, coding, mathematical or logical reasoning . |
| Approach: | They propose to use GaRAGe to evaluate whether LLMs can identify relevant grounding when generating RAG answers. |
| Outcome: | The proposed model over-summarises rather than ground answers strictly on annotated relevant passages, or deflects when no relevant grounding is available. |